support for PyCharm test debugging, and add tox environment setup #5189

gaborbernat · 2018-06-10T12:52:16Z

Most open source projects use tox as a way of automating project setup. It means the user can just invoke tox and that will setup automatically environments in what to develop and run tests. Additionally, PyCharm is a highly popular IDE, that allows users not that friendly to the command line to quickly become productive.

This PR makes some changes to ensure that out of box clone of mypy allows running the tests from within PyCharm (and that they pass). Plus, it exposes targets that also run inside the CI locally (e.g. running tests for Python 3.4, 3.5, 3.6, 3.7).

Once you have tox installed on a system or user level (pip install tox):

cd mypy
tox -av
default environments:
py34 -> run the test driver with python3.4
py35 -> run the test driver with python3.5
py36 -> run the test driver with python3.6
py37 -> run the test driver with python3.7
lint -> check the code style
type -> type check ourselves
docs -> invoke sphinx-build to build the HTML docs

additional environments:
dev  -> generate a DEV environment, that has all project libraries

E.g. to generate the documentation locally all one needs to do is tox -e docs, and can then view it by opening the HTML up from a browser inside .tox/docs_out folder. Tox is kinda like make, but cross-platform (https://tox.readthedocs.org), written in Python and built-in isolation and virtualenv creation.

These are most of the changes, I've did to help me to develop the feature #5139. Ideally this would be merged before that, leaving the PR to contain the feature only.

gvanrossum

I am unhappy with the changes to support tox. I don't want us to have to maintain another config file that we don't use in our own workflow.

gvanrossum · 2018-06-10T15:15:15Z

pytest.ini

@@ -19,5 +19,3 @@ python_files = test*.py
 python_classes =
 python_functions =

-# always run in parallel (requires pytest-xdist, see test-requirements.txt)
-addopts = -nauto --cov-append --cov-report=


Why are you deleting this? It serves a real purpose for our typical workflow (running pytest from the command line). We're trying to get rid of runtests.py, so moving it there is not a good development IMO.

@gvanrossum this is my proposal; feedback is welcome. I feel tox fills in a void that is not solved yet, so I'm not sure what type of configs are you talking about here yet; feel free to enlighten me!

This config line contains two things set, which I disabled:

run tests in parallel. Generally, I think tests should only be run in parallel when you want to run the entire test suite. E.g. becomes cumbersome to fix/check a single test with this. Shouldn't this be the other way around? That is single threaded by default, multi-threaded on demand?

automatically enable coverage report. Again this seems unwarranted when running a single test (e.g. developing a new feature, writing a new test, debugging a test). Also once this is set PyCharm debugger cannot connect anymore, as both use the same mechanism to intercept calls.

I don't want to break your workflow. As pointed out in my other PR, mainly I did not make this PR earlier because I did not know what that workflow is, and did not want to break it.

I think it would be better to document that the tests run in parallel, and how to disable that (I think setting $PYTEST_ADDOPTS to -n0 does it). Though I do agree that you probably do not want to run coverage in the general case.

gvanrossum · 2018-06-10T17:05:17Z

Since you are in the minority (none of the other devs use PyCharm to develop mypy AFAIK) don't you think it behooves you to find ways to turn off the flags you don't want rather than changing the defaults for the rest of us? Honestly I don't care about the coverage, we turned that off in Travis anyways because it's too slow. But I do care about the parallel option. My problem with supporting tox is that surely once there is a tox file we'll get PRs from others to change the tox flags and that just adds to the review load. Then again this argument also takes time so if one of the other devs speaks up I'll gladly back off. :-)

ilevkivskyi · 2018-06-10T17:27:02Z

Then again this argument also takes time so if one of the other devs speaks
up I'll gladly back off. :-)

I am using PyCharm to develop mypy approximately half of time since beginning of this year. But TBH I don't like these changes. IMO our top level directory has already too many config-related things, adding tox.ini will make it only worse. Also I don'y see much value in testing on many Python versions, the situations where something doesn't work on a given version are quite rare, and Travis will catch them (I have Travis enabled on my fork, so I see failures before even opening a PR).

gaborbernat · 2018-06-10T17:41:04Z

I had the impression people use the test runner. I made the change so that with the test runner nothing changes, moved over the config args. I did not knew people use pytest on its own.

…

On Sun, Jun 10, 2018, 18:27 Ivan Levkivskyi ***@***.***> wrote: Then again this argument also takes time so if one of the other devs speaks up I'll gladly back off. :-) I am using PyCharm to develop mypy approximately half of time since beginning of this year. *But* TBH I don't like these changes. IMO our top level directory has already too many config-related things, adding tox.ini will make it only worse. Also I don'y see much value in testing on many Python versions, the situations where something doesn't work on a given version are quite rare, and Travis will catch them (I have Travis enabled on my fork, so I see failures before even opening a PR). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5189 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAqIPhEtrfB_GB3Y7Ax7Rj8jeABsKaU8ks5t7VbrgaJpZM4UhtBv> .

gaborbernat · 2018-06-10T18:09:57Z

Oh and both setup.cfg and pytest.ini can be merged into tox.ini. And soon moved into pyproject.toml file. To help with clutter of files at root level.

emmatyping

I left a few minor points of feedback, I'll have more time tomorrow to do a more thorough review.

emmatyping · 2018-06-11T09:14:15Z

tox.ini

+[testenv]
+description = run the test driver with {basepython}
+deps = -r ./test-requirements.txt
+commands = python runtests.py -x lint -x package {posargs}


package has been replaced with self-check.

emmatyping · 2018-06-11T09:14:55Z

tox.ini

+[testenv:type]
+description = type check ourselves
+basepython = python3.6
+commands = python runtests.py testselfcheck -p '-n0' -p '-v'


self-check would be more consistent here.

emmatyping · 2018-06-11T09:16:23Z

mypy/test/testcmdline.py

@@ -58,6 +58,14 @@ def test_python_cmdline(testcase: DataDrivenTestCase) -> None:
    outb = process.stdout.read()
    # Split output into lines.
    out = [s.rstrip('\n\r') for s in str(outb, 'utf8').splitlines()]
+
+    is_running_in_py_charm = "PYCHARM_HOSTED" in os.environ
+    if is_running_in_py_charm:


I don't think you need to create a variable here. I'd just do if "PYCHARM_HOSTED" in os.environ: ...

gaborbernat · 2018-06-11T10:31:03Z

@ethanhs addressed all requests

gaborbernat · 2018-06-11T11:15:02Z

@ilevkivskyi

Also I don'y see much value in testing on many Python versions, the situations where something doesn't work on a given version are quite rare, and Travis will catch them (I have Travis enabled on my fork, so I see failures before even opening a PR).

I don't run all tests either locally. However, when Travis catches something often (especially when you're new to the code base) it helps a lot to be able to reproduce locally quickly, debug it, understand why the problem arises and being able to fix/test it locally. I suppose at the end of the day it flows into the category of different workflow between us. I had a few such instances when creating #5139.

That being said, I don't want to propose anything that breaks the current core developers workflow. I'm just suggesting to extend things that potentially help new people interested in contributing.

gaborbernat · 2018-06-12T11:53:26Z

I've restored the parallel execution of the test suite via pytest.ini and added a section on the wiki about options on how to make PyCharm play nicely with mypy (https://github.com/python/mypy/wiki/Getting-Started-Developing#pycharm). Is this more agreeable for the core developers?

I still propose keeping the tox.ini file around. That could replace the long make files from within docs. E.g. all users need to generate documentation is pip install --user tox && tox -e docs (which then will go ahead setup a valid environment and generate the documentation).

gvanrossum

OK, I am fine with this. I don't expect I'll be switching to tox myself but I don't want to stand in the way of progress for others' workflow.

emmatyping · 2018-06-13T02:16:22Z

Taking another look at this now.

gaborbernat · 2018-06-13T14:05:53Z

cool 😎 is anyone responsible with merging it?

…breaks Windows

emmatyping · 2018-06-13T17:57:59Z

I think I will defer to @ilevkivskyi since he also uses PyCharm on occasion, and this could affect his workflow.

ilevkivskyi

Just one comment, otherwise looks good to me.

ilevkivskyi · 2018-06-14T09:53:45Z

mypy/test/testcmdline.py

+        pos = next((p for p, i in enumerate(out) if i.startswith('pydev debugger: ')), None)
+        if pos is not None:
+            del out[pos]  # the attaching debugger message itself
+            del out[pos]  # plus the extra new line added


This code looks quite fragile (what if the debugger output format changes?). Is it possible to configure the debugger to just not print these lines?

Also please don't rebase your PR, merge when needed, otherwise it is hard to follow.

we'll need to follow up then and amend it; I haven't found a way to disable it sadly 😢

gaborbernat changed the title ~~adopt~~ support for PyCharm test debugging, and add tox environment setup Jun 10, 2018

gaborbernat mentioned this pull request Jun 10, 2018

option to merge stubs into source trees #5139

Closed

gvanrossum reviewed Jun 10, 2018

View reviewed changes

emmatyping requested changes Jun 11, 2018

View reviewed changes

gvanrossum approved these changes Jun 12, 2018

View reviewed changes

emmatyping approved these changes Jun 13, 2018

View reviewed changes

gaborbernat and others added 8 commits June 13, 2018 15:59

add PyCharm debugging support

ed5616b

make sure tests pass running from within PyCharm

2bebcac

add tox configuration, and expose test runner via that

8ccac35

Avoid having a space between requirement files in tox config as that …

ffc6436

…breaks Windows

Avoid redundant local variable for PyCharm identification

c3d2c28

Package suite moved to self-check

ea29148

For type tox environment use self-check package instead of the test file

f3d718d

restore pytest.ini parallel mode

e5567f4

ilevkivskyi reviewed Jun 14, 2018

View reviewed changes

ilevkivskyi approved these changes Jun 14, 2018

View reviewed changes

ilevkivskyi merged commit 5959e1a into python:master Jun 14, 2018

gaborbernat deleted the pycharm branch June 14, 2018 16:18

gaborbernat mentioned this pull request Mar 27, 2021

Add a tox.ini to make running all tests easier python/typeshed#1586

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support for PyCharm test debugging, and add tox environment setup #5189

support for PyCharm test debugging, and add tox environment setup #5189

gaborbernat commented Jun 10, 2018

gvanrossum left a comment

gvanrossum Jun 10, 2018

gaborbernat Jun 10, 2018 •

edited

Loading

emmatyping Jun 11, 2018

gvanrossum commented Jun 10, 2018 via email

ilevkivskyi commented Jun 10, 2018

gaborbernat commented Jun 10, 2018 via email

gaborbernat commented Jun 10, 2018

emmatyping left a comment

emmatyping Jun 11, 2018

emmatyping Jun 11, 2018

emmatyping Jun 11, 2018

gaborbernat commented Jun 11, 2018

gaborbernat commented Jun 11, 2018

gaborbernat commented Jun 12, 2018

gvanrossum left a comment

emmatyping commented Jun 13, 2018

gaborbernat commented Jun 13, 2018

emmatyping commented Jun 13, 2018

ilevkivskyi left a comment

ilevkivskyi Jun 14, 2018

ilevkivskyi Jun 14, 2018

gaborbernat Jun 14, 2018

ilevkivskyi Jun 14, 2018

support for PyCharm test debugging, and add tox environment setup #5189

support for PyCharm test debugging, and add tox environment setup #5189

Conversation

gaborbernat commented Jun 10, 2018

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaborbernat Jun 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum commented Jun 10, 2018 via email

ilevkivskyi commented Jun 10, 2018

gaborbernat commented Jun 10, 2018 via email

gaborbernat commented Jun 10, 2018

emmatyping left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaborbernat commented Jun 11, 2018

gaborbernat commented Jun 11, 2018

gaborbernat commented Jun 12, 2018

gvanrossum left a comment

Choose a reason for hiding this comment

emmatyping commented Jun 13, 2018

gaborbernat commented Jun 13, 2018

emmatyping commented Jun 13, 2018

ilevkivskyi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaborbernat Jun 10, 2018 •

edited

Loading